Methods and apparatus to specify regions of interest in video frames

ABSTRACT

Methods and apparatus to specify regions of interest in video frames are disclosed. Example disclosed methods to mark a region in a graphical presentation include selecting a first point located at a substantially central position within the region, selecting a plurality of second points to define a boundary of the region, and comparing a plurality of stored templates with the selected first and second points to identify a first one of the stored templates to represent the region.

RELATED APPLICATION(S)

This patent arises from a continuation of U.S. patent application Ser.No. 13/250,331, entitled “Methods and Apparatus to Specify Regions ofInterest in Video Frames,” which was filed on Sep. 30, 2011, which is acontinuation of U.S. patent application Ser. No. 12/239,425 (now U.S.Pat. No. 8,059,865), entitled “Methods and Apparatus to Specify Regionsof Interest in Video Frames,” which was filed on Sep. 26, 2008, whichclaims priority to U.S. Provisional Application Ser. No. 60/986,723,entitled “Methods and Apparatus to Measure Brand Exposure in MediaStreams,” which was filed on Nov. 9, 2007. U.S. patent application Ser.No. 13/250,331, U.S. patent application Ser. No. 12/239,425 and U.S.Provisional Application Ser. No. 60/986,723 are hereby incorporated byreference in their respective entireties.

FIELD OF THE DISCLOSURE

This disclosure relates generally to video frame processing, and, moreparticularly, to methods and apparatus to specify regions of interest invideo frames.

BACKGROUND

As used herein, a “broadcast” refers to any sort of electronictransmission of any sort of media signal(s) from a source to one or morereceiving devices of any kind. Thus, a “broadcast” may be a cablebroadcast, a satellite broadcast, a terrestrial broadcast, a traditionalfree television broadcast, a radio broadcast, and/or an internetbroadcast, and a “broadcaster” may be any entity that transmits signalsfor reception by a plurality of receiving devices. The signals mayinclude media content (also referred to herein as “content” or“programs”), and/or commercials (also referred to herein as“advertisements”). An “advertiser” is any entity that provides anadvertisement for broadcast. Traditionally, advertisers have paidbroadcasters to interleave commercial advertisements with broadcastcontent (e.g., in a serial“content-commercial-content-commercial”format) such that, to view anentire program of interest, the audience is expected to view theinterleaved commercials. This approach enables broadcasters to supplyfree programming to the audience while collecting fees for theprogramming from sponsoring advertisers.

To facilitate this sponsorship model, companies that rely on broadcastvideo and/or audio programs for revenue, such as advertisers,broadcasters and content providers, wish to know the size anddemographic composition of the audience(s) that consume program(s).Merchants (e.g., manufacturers, wholesalers and/or retailers) also wantto know this information so they can target their advertisements to thepopulations most likely to purchase their products. Audience measurementcompanies have addressed this need by, for example, identifying thedemographic composition of a set of statistically selected householdsand/or individuals (i.e., panelists) and the program consumption habitsof the member(s) of the panel. For example, audience measurementcompanies may collect viewing data on a selected household by monitoringthe content displayed on that household's television(s) and byidentifying which household member(s) are present in the room when thatcontent is displayed. An analogous technique is applied in the radiomeasurement context.

Gathering this audience measurement data has become more difficult asthe diversity of broadcast systems has increased. For example, while itwas once the case that television broadcasts were almost entirelyterrestrial based, radio frequency broadcast systems (i.e., traditionalfree television), cable and satellite broadcast systems have now becomecommonplace. Further, these cable and/or satellite based broadcastsystems often require the use of a dedicated receiving device such as aset top box (STB) or an integrated receiver decoder (IRD) to tune,decode, and/or display broadcast programs. To complicate mattersfurther, some of these receiving devices for alternative broadcastsystems as well as other receiving devices such as local media playbackdevices (e.g., video cassette recorders, digital video recorders, and/orpersonal video recorders) have made time shifted viewing of broadcastand other programs possible.

This ability to record and playback programming (i.e., time-shifting)has raised concerns in the advertising industry that consumers employingsuch time shifting technology will skip or otherwise fast forwardthrough commercials when viewing recorded programs, thereby underminingthe effectiveness of the traditional interleaved advertising model. Toaddress this issue, rather than, or in addition to, interleavingcommercials with content, merchants and advertisers have begun payingcontent creators a fee to place their product(s) within the contentitself. For example, a manufacturer of a product (e.g., sunglasses)might pay a content creator a fee to have their product appear in abroadcast program (e.g., to have their sunglasses worn by an actor inthe program) and/or to have their product mentioned by name during theprogram. It will be appreciated that the sunglasses example is merelyillustrative and any other product or service of interest could beintegrated into the programming in any desired fashion (e.g., if theproduct were a soft drink, an advertiser may pay a fee to have a castmember drink from a can displaying the logo of the soft drink).

Along similar lines, advertisers have often paid to place advertisementssuch as billboards, signs, etc. in locations from which broadcasting islikely to occur such that their advertisements appear in broadcastcontent. Common examples of this approach are the billboards and othersigns positioned throughout arenas used to host sporting events,concerts, political events, etc. Thus, when, for example, a baseballgame is broadcast, the signs along the perimeter of the baseball field(e.g., “Buy Sunshine Brand Sunglasses”) are likewise broadcast asincidental background to the sporting event.

Due to the placement of the example sunglasses in the program and/or dueto the presence of the example advertisement signage at the location ofthe broadcast event, the advertisement for the sunglasses and/or theadvertisement signage (collectively and/or individually referred toherein as “embedded advertisement”) is embedded in the broadcastcontent, rather than in a commercial interleaved with the content.Consequently, it is not possible for an audience member to fast forwardor skip past the embedded advertisement without also fast forwarding orskipping past a portion of the program in which the advertisement isembedded. As a result, it is believed that audience members are lesslikely to skip the advertisement and, conversely, that audience membersare more likely to view the advertisement than in the traditionalinterleaved content-commercial(s)-content-commercial(s) approach tobroadcast advertising.

The advertising approach of embedding a product in content is referredto herein as “intentional product placement,” and products placed byintentional product placement are referred to herein as “intentionallyplaced products.” It will be appreciated that content may includeintentionally placed products (i.e., products that are used as props inthe content in exchange for a fee from an advertiser and/or merchant)and unintentionally placed products. As used herein, “unintentionallyplaced products” are products that are used as props in content bychoice of the content creator without payment from an advertiser ormerchant. Thus, an unintentionally placed product used as a prop iseffectively receiving free advertisement, but may have been included forthe purpose of, for example, story telling and not for the purpose ofadvertising.

Similarly, the advertising approach of locating a sign, billboard orother display advertisement at a location where it is expected to beincluded in a broadcast program such as a sporting event is referred toherein as “intentional display placement,” and advertising displays ofany type which are placed by intentional display placement are referredto herein as “intentionally placed displays.” It will be appreciatedthat content may include intentionally placed displays (i.e., displaysthat were placed to be captured in a broadcast) and unintentionallyplaced displays (i.e., displays that are not intended by the advertiserto be captured in content, but, due to activity by a content creator,they are included incidentally in the content through, for example,filming a movie or television show in Times Square, filming a live newsstory on a city street adjacent a billboard or store front sign, etc.).Additionally, as used herein “intentionally placed advertisement”generically refers to any intentionally placed product and/or anyintentionally placed display. Analogously, “unintentionally placedadvertisement” generically refers to any unintentionally placed productand/or any unintentionally placed display.

The brand information (e.g., such as manufacturer name, distributorname, provider name, product/service name, catch phrase, etc.), as wellas the visual appearance (e.g., such as screen size, screen location,occlusion, image quality, venue location, whether the appearance isstatic or changing (e.g., animated), whether the appearance is real or avirtual overlay, etc.) and/or audible sound of the same included in anembedded advertisement (e.g., such as an intentional or unintentionalproduct placement, display placement or advertising placement) isreferred to herein as a “brand identifier” or, equivalently, a “logo”for the associated product and/service. For example, in the case of anintentional display placement of a sign proclaiming “Buy Sunshine BrandSunglasses” placed along the perimeter of a baseball field, the wordsand general appearance of the phrase “Buy Sunshine Brand Sunglasses”comprise the brand identifier (e.g., logo) corresponding to thisintentional display placement.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic illustration of an example system to measure brandexposure in media streams.

FIG. 2 illustrates an example manner of implementing the example brandexposure monitor of FIG. 1.

FIG. 3 illustrates an example manner of implementing the example scenerecognizer of FIG. 2.

FIG. 4 illustrates an example manner of implementing the example brandrecognizer of FIG. 2.

FIGS. 5A-5D illustrate example scene classifications made by the examplescene recognizer of FIG. 2.

FIGS. 6A-6B collectively from a flowchart representative of examplemachine accessible instructions that may be executed to implement theexample scene recognizer of FIG. 2.

FIGS. 7A-7C collectively from a flowchart representative of examplemachine accessible instructions that may be executed to implement theexample graphical user interface (GUI) of FIG. 2.

FIGS. 8A-8B are flowcharts representative of example machine accessibleinstructions that may be executed to implement the example brandrecognizer of FIG. 2.

FIG. 9 is a schematic illustration of an example processor platform thatmay be used to execute some or all of the machine accessibleinstructions of FIGS. 6A-6B, 7A-7C and/or 8A-8B to implement the methodsand apparatus described herein.

FIG. 10 illustrates an example sequence of operations performed by anexample automated region of interest creation technique that may be usedto implement the methods and apparatus described herein.

DETAILED DESCRIPTION

The terms “brand exposure” and “exposures to brand identifiers” as usedherein refer to the presentation of one or more brand identifiers inmedia content delivered by a media content stream, thereby providing anopportunity for an observer of the media content to become exposed tothe brand identifier(s) (e.g., logo(s)). As used herein, a brandexposure does not require that the observer actually observe the brandidentifier in the media content, but instead indicates that the observerhad an opportunity to observe the brand identifier, regardless ofwhether the observer actually did so. Brand exposures may be tabulatedand/or recorded to determine the effectiveness of intentional orunintentional product placement, display placement or advertisingplacement.

In the description that follows, a broadcast of a baseball game is usedas an example of a media stream that may be processed according to themethods and/or apparatus described herein to determine brand exposure.It will be appreciated that the example baseball game broadcast ismerely illustrative and the methods and apparatus disclosed herein arereadily applicable to processing media streams to determine brandexposure associated with any type of media content. For example, themedia content may correspond to any type of sporting event, including abaseball game, as well as any television program, movie, streaming videocontent, video game presentation, etc.

FIG. 1 is a schematic illustration of an example system to measure brandexposures in media streams. The example system of FIG. 1 utilizes one ormore media measurement techniques, such as, for example, audio codes,audio signatures, video codes, video signatures, image codes, imagesignatures, etc., to identify brand exposures in presented media content(e.g., such as content currently being broadcast or previously recordedcontent) provided by one or more media streams. In an exampleimplementation, image signatures corresponding to one or more portionsof a media stream are compared with a database of reference imagesignatures that represent corresponding portions of reference mediacontent to facilitate identification of one or more scenes broadcast inthe media stream and/or one or more brand identifiers included in thebroadcast scene(s).

To process (e.g., receive, play, view, record, decode, etc.) and presentany number and/or type(s) of content, the example system of FIG. 1includes any number and/or type(s) of media device(s) 105. The mediadevice(s) 105 may be implemented by, for example, a set top box (STB), adigital video recorder (DVR), a video cassette recorder (VCR), apersonal computer (PC), a game console, a television, a media player,etc., or any combination thereof. Example media content includes, but isnot limited to, television (TV) programs, movies, videos, websites,commercials/advertisements, audio, games, etc. In the example system ofFIG. 1, the example media device 105 receives content via any numberand/or type(s) of sources such as, for example: a satellite receiverand/or antenna 110, a radio frequency (RF) input signal 115corresponding to any number and/or type(s) of cable TV signal(s) and/orterrestrial broadcast(s), any number and/or type(s) of datacommunication networks such as the Internet 120, any number and/ortype(s) of data and/or media store(s) 125 such as, for example, a harddisk drive (HDD), a VCR cassette, a digital versatile disc (DVD), acompact disc (CD), a flash memory device, etc. In the example system ofFIG. 1, the media content (regardless of its source) may include forexample, video data, audio data, image data, website data, etc.

To generate the content for processing and presentation by the examplemedia device(s) 105, the example system of FIG. 1 includes any numberand/or type(s) of content provider(s) 130 such as, for example,television stations, satellite broadcasters, movie studios, websiteproviders, etc. In the illustrated example of FIG. 1, the contentprovider(s) 130 deliver and/or otherwise provide the content to theexample media device 105 via any or all of a satellite broadcast using asatellite transmitter 135 and a satellite and/or satellite relay 140, aterrestrial broadcast received via the RF input signal 115, a cable TVbroadcast received via the RF input signal 115, the Internet 120, and/orthe media store(s) 125.

To measure brand exposure (i.e., exposures to brand identifiers) inmedia stream(s) processed and presented by the example media device(s)105, the example system of FIG. 1 includes at least one brand exposuremonitor, one of which is illustrated at reference number 150 of FIG. 1.The example brand exposure monitor 150 of FIG. 1 processes a mediastream 160 output by the example media device 105 to identify at leastone brand identifier being presented via the media device 105. Ingeneral, the example brand exposure monitor 150 operates to identifybrand identifiers and report brand exposure(s) automatically using knownor previously learned information when possible, and then defaults torequesting manual user input when such automatic identification is notpossible. At a high-level, the example brand exposure monitor 150achieves this combination of automatic and manual brand exposureprocessing by first dividing the media stream 160 into a group ofsuccessive detected scenes, each including a corresponding group ofsuccessive image frames. The example brand exposure monitor 150 thenexcludes any scenes known not to include any brand identifierinformation. Next, the example brand exposure monitor 150 compares eachnon-excluded detected scene to a library of reference scenes todetermine whether brand exposure monitoring may be performedautomatically. For example, automatic brand exposure monitoring ispossible if the detected scene matches information stored in thereference library which corresponds to a repeated scene of interest or aknown scene of no interest. However, if the detected scene does notmatch (or fully match) information in the reference library, automaticbrand exposure monitoring is not possible and the example brand exposuremonitor 150 resorts to manual user intervention to identify some or allof the brand identifier(s) included in the detected scene for brandexposure reporting.

Examining the operation of the example brand exposure monitor 150 ofFIG. 1 in greater detail, the example brand exposure monitor 150determines (e.g., collects, computes, extracts, detects, recognizes,etc.) content identification information (e.g., such as at least oneaudio code, audio signature, video code, video signature, image code,image signature, etc.) to divide the media stream 160 into a group ofsuccessive scenes. For example, the brand exposure monitor 150 maydetect a scene of the media stream 160 as corresponding to a sequence ofadjacent video frames (i.e., image frames) having substantially similarcharacteristics such as, for example, a sequence of frames correspondingto substantially the same camera parameters (e.g., angle, height,aperture, focus length, etc.) and having background that isstatistically stationary (e.g., the background may have individualcomponents that move, but the overall background on average appearsrelative stationary). The brand exposure monitor 150 of the illustratedexample utilizes scene change detection to mark the beginning imageframe and the ending image frame corresponding to a scene. In an exampleimplementation, the brand exposure monitor 150 performs scene changedetection by creating an image signature for each frame of the mediastream 160 (possibly after subsampling) and then comparing the imagesignatures of a sequence of frames to determine when a scene changeoccurs. For example, the brand exposure monitor 150 may compare theimage signature corresponding to the starting image of a scene to theimage signatures for one or more successive image frames following thestarting frame. If the image signature for the starting frame does notdiffer significantly from a successive frame's image signature, thesuccessive frame is determined to be part of the same scene as thestarting frame. However, if the image signatures are found to differsignificantly, the successive frame that differs is determined to be thestart of a new scene and becomes the first frame for that new scene.Using the example of a media stream 160 providing a broadcast of abaseball game, a scene change occurs when, for example, the videoswitches from a picture of a batter to a picture of the outfield afterthe batter successfully hits the ball.

Next, after the scene is detected, the brand exposure monitor 150 of theillustrated example determines at least one key frame and key imagesignature representative of the scene. For example, the key frame(s) andkey image signature(s) for the scene may be chosen to be the frame andsignature corresponding to the first frame in the scene, the last framein the scene, the midpoint frame in the scene, etc. In another example,the key frame(s) and key image signature(s) may be determined to be anaverage and/or some other statistical combination of the frames and/orsignatures corresponding to the detected scene.

To reduce processing requirements, the brand exposure monitor 150 mayexclude a detected scene under circumstances where it is likely thescene will not contain any brand identifiers (e.g., logos). In anexample implementation, the brand exposure monitor 150 is configured touse domain knowledge corresponding to the particular type of mediacontent being processed to determine when a scene exhibitscharacteristics indicating that the scene will not contain any brandidentifiers. For example, in the context of media content correspondingto the broadcast of a baseball game, a scene including a backgrounddepicting only the turf of the baseball field may be known not tocontain any brand identifiers. In such a case, the brand exposuremonitor 150 may exclude a scene from brand exposure monitoring if thescene exhibits characteristics of a scene depicting the turf of thebaseball field (e.g., such as a scene having a majority of pixels thatare predominantly greenish in color and distributed such that, forexample, the top and bottom areas of the scene include regions ofgreenish pixels grouped together). If a detected scene is excluded, thebrand exposure monitor 150 of the illustrated example reports theexcluded scene and then continues processing to detect the next scene ofthe media stream 160.

Assuming that a detected scene is not excluded, the example brandexposure monitor 150 then compares the image signature for the detectedscene with one or more databases (not shown) of reference signaturesrepresentative of previously learned and/or known scenes to determinewhether the current detected scene is a known scene or a new scene. Ifthe current scene matches a previously learned and/or known scene storedin the database(s), the brand exposure monitor 150 obtains statusinformation for the scene from the database(s). If the statusinformation indicates that the scene had been previously marked as ascene of no interest in the database(s) (e.g., such as a scene known notto include any brand identifiers (e.g., logos)), the scene is reportedas a scene of no interest and may be included in the database(s) aslearned information to be used to identify future scenes of no interest.For example, and as discussed in greater detail below, a scene may bemarked as a scene of no interest if it is determined that no brandidentifiers (e.g., logos) are visible in the scene. The brand exposuremonitor 150 of the illustrated example then continues processing todetect the next scene of the media stream 160.

If, however, the current scene is indicated to be a scene of interest,the brand exposure monitor 150 then determines one or more expectedregions of interest residing within the current scene that may contain abrand identifier (e.g., logo), as discussed in greater detail below. Thebrand exposure monitor 150 then verifies the expected region(s) ofinterest with one or more databases (not shown) storing informationrepresentative of reference (e.g., previously learned and/or known)brand identifiers (e.g., logos). If all of the expected region(s) ofinterest are verified to include corresponding expected brandidentifier(s), the example brand exposure monitor 150 reports exposureto matching brand identifiers.

However, if the current scene does not match any reference (e.g.,previously learned and/or known) scene, and/or at least one region ofinterest does not match one or more reference (e.g., previously learnedand/or known) brand identifiers, the brand exposure monitor 150initiates a graphical user interface (GUI) session at the GUI 152. TheGUI 152 is configured to display the current scene and prompt the user170 to provide an identification of the scene and/or the brandidentifiers included in the region(s) of interest. For each brandidentifier recognized automatically or via information input by the user170 via the GUI 152, corresponding data and/or reports are stored in anexample brand exposure database 155 for subsequent processing. After thescene and/or brand identifier(s) have been identified by the user 170via the GUI 152, the current scene and/or brand identifier(s) are storedin their respective database(s). In this way, the current scene and/orbrand identifier(s), along with any corresponding descriptiveinformation, are learned by the brand exposure monitor 150 and can beused to detect future instances of the scene and/or brand identifier(s)in the media stream 160 without further utilizing the output deviceand/or GUI 152. An example manner of implementing the example brandexposure monitor 150 of FIG. 1 is described below in connection withFIG. 2.

During scene and/or brand identifier recognition, the brand exposuremonitor 150 may also present any corresponding audio content to the user170 to further enable identification of any brand audio mention(s). Upondetection of an audio mention of a brand, the user 170 may so indicatethe audio mention to the brand exposure monitor 150 by, for example,clicking on an icon on the GUI 152, inputting descriptive informationfor the brand identifier (e.g., logo), etc. Furthermore, key words fromclosed captioning, screen overlays, etc., may be captured and associatedwith detected audio mentions of the brand. Additionally oralternatively, audio, image and/or video codes inserted by contentproviders 130 to identify content may be used to identify brandidentifiers. For example, an audio code for a segment of audio of themedia stream 160 may be extracted and cross-referenced to a database ofreference audio codes. Audio exposure of a detected brand identifier mayalso be stored in the example brand exposure database 155. The audiomentions stored in the example brand exposure database 155 may alsocontain data that links the audio mention(s) to scene(s) beingbroadcast. Additionally, the identified audio mentions may be added toreports and/or data regarding brand exposure generated from the examplebrand exposure database 155.

To record information (e.g., such as ratings information) regardingaudience consumption of the media content provided by the media stream160, the example system of FIG. 1 includes any number and/or type(s) ofaudience measurements systems, one of which is designated at referencenumeral 180 in FIG. 1. The example audience measurement system 180 ofFIG. 1 records and/or stores in an example audience database 185information representative of persons, respondents, households, etc.,consuming and/or exposed to the content provided and/or delivered by thecontent providers 130. The audience information and/or data stored inthe example audience database 185 may be further combined with the brandexposure information/data recorded in the example brand exposuredatabase 155 by the brand exposure monitor 150. In the illustratedexample, the combined audience/brand exposure information/data is storedin an example audience and brand exposure database 195. The combinationof audience information and/or data and brand based exposure measurementinformation 195 may be used, for example, to determine and/or estimateone or more statistical values representative of the number of personsand/or households exposed to one or more brand identifiers.

FIG. 2 illustrates an example manner of implementing the example brandexposure monitor 150 of FIG. 1. To process the media stream 160 of FIG.1, the brand exposure monitor 150 of FIG. 2 includes a scene recognizer252. The example scene recognizer 252 of FIG. 2 operates to detectscenes and create one or more image signatures for each identified sceneincluded in the media stream 160. In an example implementation, themedia stream 160 includes a video stream comprising a sequence of imageframes having a certain frame rate (e.g., such as 30 frames per second).A scene corresponds to a sequence of adjacent image frames havingsubstantially similar characteristics. For example, a scene correspondsto a sequence of images captured with similar camera parameters (e.g.,angle, height, aperture, focus length) and having a background that isstatistically stationary (e.g., the background may have individualcomponents that move, but the overall background on average appearsrelatively stationary). To perform scene detection, the example scenerecognizer 252 creates an image signature for each image frame (possiblyafter sub-sampling at a lower frame rate). The example scene recognizerthen compares the image signatures within a sequence of frames to acurrent scene's key signature(s) to determine whether the imagesignatures are substantially similar or different. As discussed above, acurrent scene may be represented by one or more key frames (e.g., suchas the first frame, etc.) with a corresponding one or more keysignatures. If the image signatures for the sequence of frames aresubstantially similar to the key signature, the image frames areconsidered as corresponding to the current scene and at least one of theframes in the sequence (e.g., such as the starting frame, the midpointframe, the most recent frame, etc.) is used as a key frame to representthe scene. The image signature corresponding to the key frame is thenused as the image signature for the scene itself. If, however, a currentimage signature corresponding to a current image frame differssufficiently from the key frames signature(s), the current image framecorresponding to the current image signature is determined to mark thestart of a new scene of the media stream 160. Additionally, the mostrecent previous frame is determined to mark the end of the previousscene.

How image signatures are compared to determine the start and end framesof scenes of the media stream 160 depends on the characteristics of theparticular image signature technique implemented by the scene recognizer252. In an example implementation, the scene recognizer 252 creates ahistogram of the luminance (e.g., Y) and chrominance (e.g., U & V)components of each image frame or one or more specified portions of eachimage. This image histogram becomes the image signature for the imageframe. To compare the image signatures of two frames, the example scenerecognizer 252 performs a bin-wise comparison of the image histogramsfor the two frames. The scene recognizer 252 then totals the differencesfor each histogram bin and compares the computed difference to one ormore thresholds. The thresholds may be preset and/or programmable, andmay be tailored to balance a trade-off between scene granularity vs.processing load requirements.

The scene recognizer 252 of the illustrated example may also implementscene exclusion to further reduce processing requirements. As discussedabove, the example scene recognizer 252 may exclude a scene based on,for example, previously obtained domain knowledge concerning the mediacontent carried by the example media stream 160. The domain knowledge,which may or may not be unique to the particular type of media contentbeing processed, may be used to create a library of exclusioncharacteristics indicative of a scene that will not include any brandidentifiers (e.g., logos). If scene exclusion is implemented, theexample scene recognizer 252 may mark a detected scene for exclusion ifit possesses some or all of the exclusion characteristics. For example,and as discussed above, in the context of media content corresponding tothe broadcast of a baseball game, a scene characterized by apredominantly greenish background may be marked for exclusion becausethe scene corresponds to a camera shot depicting the turf of thebaseball field. This is because, based on domain knowledge concerningbroadcasted baseball games, it is known that camera shots of thebaseball field's turf rarely, if ever, include any brand identifiers tobe reported. As discussed above, the example scene recognizer 252 of theillustrated example reports any excluded scene and then continuesprocessing to detect the next scene of the media stream 160.Alternatively, the example scene recognizer 252 could simply discard theexcluded scene and continue processing to detect the next scene of themedia stream 160. An example manner of implementing the example scenerecognizer 252 of FIG. 2 is discussed below in connection with FIG. 3.

Assuming that the detected scene currently being processed (referred toas the “current scene”) is not excluded, the example scene recognizer252 begins classifying the scene into one of the following fourcategories: a repeated scene of interest, a repeated scene of changedinterest, a new scene, or a scene of no interest. For example, a sceneof no interest is a scene known or previously identified as including novisible brand identifiers (e.g., logos). A repeated scene of interest isa scene of interest known to include visible brand identifiers (e.g.,logos) and in which all visible brand identifiers are already known andcan be identified. A repeated scene of changed interest is a scene ofinterest known to include visible brand identifiers (e.g., logos) and inwhich some visible brand identifiers (e.g., logos) are already known andcan be identified, but other visible brand identifiers are unknownand/or cannot be identified automatically. A new scene corresponds to anunknown scene and, therefore, it is unknown whether the scene includesvisible brand identifiers (e.g., logos).

To determine whether the current scene is a scene of no interest orwhether the scene is one of the other scenes of interest that maycontain visible brand identifiers, the example scene recognizer 252compares the image signature for the scene with one or more referencesignatures. The reference signatures may correspond to previously knownscene information stored in a scene database 262 and/or previouslylearned scene information stored in a learned knowledge database 264. Ifthe current scene's image signature does not match any of the availablereference signatures, the example scene recognizer 252 classifies thescene as a new scene. If the scene's image signature does match one ormore of the available reference signatures, but information associatedwith the matched reference signature(s) and stored in the scene database262 and/or learned knowledge database 264 indicates that the sceneincludes no visible brand identifiers, the example scene recognizer 252classifies the scene as a scene of no interest. Otherwise, the scenewill be a classified as either a repeated scene of interest or arepeated scene of changed interest by the example scene recognizer 252as discussed below.

In an example implementation using the image histograms described aboveto represent image signatures, a first threshold (or first thresholds)could be used for scene detection, and a second threshold (or secondthresholds) could be used for scene classification based on comparisonwith reference scenes. In such an implementation, the first threshold(s)would define a higher degree of similarity than the second threshold(s).In particular, while the first threshold(s) would define a degree ofsimilarity in which there was little or no change (at leaststatistically) between image frames, the second threshold(s) woulddefine a degree of similarity in which, for example, some portions ofthe compared frames could be relatively similar, whereas other portionscould be different. For example, in the context of the broadcastbaseball game example, a sequence of frames showing a first batterstanding at home plate may meet the first threshold(s) such that all theframes in the sequence are determined to belong to the same scene. Whena second batter is shown standing at home plate, a comparison of firstframe showing the second batter with the frame(s) showing the firstbatter may not meet the first threshold(s), thereby identifying thestart of a new scene containing the second batter. However, because thebackground behind home plate will be largely unchanging, a comparison ofthe first scene containing the first batter with the second scenecontaining the second batter may meet the second threshold(s),indicating that the two scenes should be classified as similar scenes.In this particular example, the scene containing the second batter wouldbe considered a repeated scene relative to the scene containing thefirst batter.

The scene database 262 may be implemented using any data structure(s)and may be stored in any number and/or type(s) of memories and/or memorydevices 260. The learned knowledge database 264 may be implemented usingany data structure(s) and may be stored in any number and/or type(s) ofmemories and/or memory devices 260. For example, the scene database 262and/or the learned knowledge database 264 may be implemented usingbitmap files, a JPEG file repository, etc.

To determine whether the scene having a signature matching one or morereference signatures is a repeated scene of interest or a repeated sceneof changed interest, the example scene recognizer 252 of FIG. 2identifies one or more expected regions of interest included in thescene at issue based on stored information associated with referencescene(s) corresponding to the matched reference signature(s). An examplebrand recognizer 254 (also known as a logo detector 254) included in theexample brand exposure monitor 150 then performs brand identifierrecognition (also known as “logo detection”) by comparing and verifyingthe expected region(s) of interest with information corresponding to oneor more corresponding expected reference brand identifiers stored in thelearned knowledge database 264 and/or a brand library 266. For example,the brand recognizer 254 may verify that the expected reference brandidentifier(s) is/are indeed included in the expected region(s) ofinterest by comparing each expected region of interest in the scene'skey frame with known brand identifier templates and/or templates storedin the example learned knowledge database 264 and/or a brand library266. The brand library 266 may be implemented using any datastructure(s) and may be stored in any number and/or type(s) of memoriesand/or memory devices 260. For example, the brand library 266 may storethe information in a relational database, a list of signatures, a bitmapfile, etc. Example techniques for brand identifier recognition (or logodetection) are discussed in greater detail below.

Next, for each verified region of interest, the example brand recognizer254 initiates a tracker function to track the contents of the verifiedregion of interest across all the actual image frames include in thecurrent scene. For example, the tracker function may compare aparticular region of interest in the current scene's key frame withcorresponding region(s) of interest in each of the other frames in thecurrent scene. If the tracker function verifies that the correspondingexpected region(s) of interest match in all of the current scene's imageframes, the example scene recognizer 252 classifies the scene as arepeated scene of interest. If, however, at least one region of interestin at least one of the current scene's image frames could not beverified with a corresponding expected reference brand identifier, thescene recognizer 252 classifies the scene as a repeated scene of changedinterest. The processing of repeated scenes of changed interest isdiscussed in greater detail below. After classification of the scene,the scene recognizer 252 continues to detect and/or classify the nextscene in the media stream 160.

To provide identification of unknown and/or unidentified brandidentifiers included in new scenes and repeated scenes of changedinterest, the example brand exposure monitor 150 of FIG. 1 includes theGUI 152. The example GUI 152, also illustrated in FIG. 2, displaysinformation pertaining to the scene and prompts the user 170 to identifyand/or confirm the identity of the scene and/or one or more potentialbrand identifiers included in one or more regions of interest. Theexample GUI 152 may be displayed via any type of output device 270, suchas a television (TV), a computer screen, a monitor, etc., when a newscene or a repeated scene of changed interest is identified by theexample scene recognizer 252. In an example implementation, when a sceneis classified as a new scene or a repeated scene of changed interest,the example scene recognizer 252 stops (e.g., pauses) the media stream160 of FIG. 1 and then the GUI 152 prompts the user 170 foridentification of the scene and/or identification of one or more regionsof interest and any brand identifier(s) included in the identifiedregion(s) of interest. For example, the GUI 152 may display a blankfield to accept a scene name and/or information regarding a brandidentifier provided by the user 170, provide a pull down menu ofpotential scene names and/or brand identifiers, suggest a scene nameand/or a brand identifier which may be accepted and/or overwritten bythe user 170, etc. To create a pull down menu and/or an initial value tobe considered by the user 170 to identify the scene and/or any brandidentifiers included in any respective region(s) of interest of thescene, the GUI 152 may obtain data stored in the scene database 262, thelearned knowledge database 264 and/or the brand library 266.

To detect the size and shape of one or more regions of interest includedin a scene, the example GUI 152 of FIG. 2 receives manual input tofacilitate generation and/or estimation of the location, boundariesand/or size of each region of interest. For example, the example GUI 152could be implemented to allow the user 170 to mark a given region ofinterest by, for example, (a) clicking on one corner of the region ofinterest and dragging the cursor to the furthest corner, (b) placing thecursor on each corner of the region of interest and clicking while thecursor is at each of the corners, (c) clicking anywhere in the region ofinterest with the GUI 152 estimating the size and/or shape to calculateof the region of interest, etc.

Existing techniques for specifying and/or identifying regions ofinterest in video frames typically rely on manually marked regions ofinterest specified by a user. Many manual marking techniques require auser to carefully mark all vertices of a polygon bounding a desiredregion of interest, or otherwise carefully draw the edges of some otherclosed graphical shape bounding the region of interest. Such manualmarking techniques can require fine motor control and hand-eyecoordination, which can result in fatigue if the number of regions ofinterest to be specified is significant. Additionally, different userare likely to mark regions of interest differently using existing manualmarking techniques, which can result in irreproducible, imprecise and/orinconsistent monitoring performance due to variability in thespecification of regions of interest across the video frames associatedwith the broadcast content undergoing monitoring.

In a first example region of interest marking technique that may beimplemented by the example GUI 152, the GUI 152 relies on manual markingof the perimeter of a region of interest. In this first example markingtechnique, the user 170 uses a mouse (or any other appropriate inputdevice) to move a displayed cursor to each point marking the boundary ofthe desired region of interest. The user 170 marks each boundary pointby clicking a mouse button. After all boundary points are marked, theexample GUI 152 connects the marked points in the order in which theywere marked, thereby forming a polygon (e.g., such as a rectangle)defining the region of interest. Any area outside the polygon isregarded as being outside the region of interest. As mentioned above,one potential drawback of this first example region of interest markingtechnique is that manually drawn polygons can be imprecise andinconsistent. This potential lack of consistency can be especiallyproblematic when region(s) of interest for a first set of scenes aremarked by one user 170, and region(s) of interest from some second setof scenes are marked by another user 170. For example, inconsistenciesin the marking of regions of interest may adversely affect the accuracyor reliability of any matching algorithms/techniques relying on themarked regions of interest.

In a second example region of interest marking technique that may beimplemented by the example GUI 152, the GUI 152 implements a moreautomatic and consistent approach to marking a desired region ofinterest in any type of graphical presentation. For example, this secondexample region of interest marking technique may be used to mark adesired region of interest in an image, such as corresponding to a videoframe or still image. Additionally or alternatively, the second exampleregion of interest marking technique may be used to mark a desiredregion of interest in a drawing, diagram, slide, poster, table,document, etc., created using, for example, any type of computer aideddrawing and/or drafting application, word processing application,presentation creation application, etc. The foregoing example ofgraphical presentations are merely illustrative and are not meant to belimiting with respect to the type of graphical presentations for whichthe second example region of interest marking technique may be used tomark a desired region of interest.

In this automated example region of interest marking technique, the user170 can create a desired region of interest from scratch or based on astored and/or previously created region of interest acting as atemplate. An example sequence of operations to create a region ofinterest from scratch using this example automated region of interestmarking technique is illustrated in FIG. 10. Referring to FIG. 10, tocreate a region of interest in an example scene 1000 from scratch, theuser 170 uses a mouse (or any other appropriate input device) to clickanywhere inside the desired region of interest to create a referencepoint 1005. Once the reference point 1005 is marked, the example GUI 152determines and displays an initial region 1010 around the referencepoint 1005 to serve as a template for region of interest creation.

In an example implementation, the automated region of interest markingtechnique illustrated in FIG. 10 compares adjacent pixels in a recursivemanner to automatically generate the initial template region 1010. Forexample, starting with the initially selected reference point 1005,adjacent pixels in the four directions of up, down, left and right arecompared to determine if they are similar (e.g., in luminance andchrominance) to the reference point 1005. If any of these four adjacentpixels are similar, each of those similar adjacent pixels then forms thestarting point for another comparison in the four directions of up,down, left and right. This procedure continues recursively until nosimilar adjacent pixels are found. When no similar adjacent pixels arefound, the initial template region 1010 is determined to be a polygon(e.g., specified by vertices, such as a rectangle specified by fourvertices) or an ellipse (e.g., specified by major and minor axes)bounding all of the pixels recursively found to be similar to theinitial reference point 1005. As an illustrative example, in FIG. 10 thereference point 1005 corresponds to a position on the letter “X”(labeled with reference numeral 1015) as shown. Through recursive pixelcomparison, all of the dark pixels comprising the letter “X” (referencenumeral 1015) will be found to be similar to the reference point 1005.The initial template region 1010 is then determined to be a rectangularregion bounding all of the pixels recursively found to be similar in theletter “X” (reference numeral 1015).

The automated region of interest marking technique illustrated in FIG.10 can also automatically combine two or more initial template regionsto create a single region of interest. As an illustrative example, inFIG. 10, as discussed above, selecting the reference point 1005 causesthe initial template region 1010 to be determined as bounding all of thepixels recursively found to be similar in the letter “X” (referencenumeral 1015). Next, if the reference point 1020 was selected, a secondinitial template region 1025 would be determined as bounding all of thepixels recursively found to be similar in the depicted letter “Y”(labeled with reference numeral 1030). After determining the first andsecond initial template regions 1010 and 1025 based on the respectivefirst and second reference points 1005 and 1020, a combined region ofinterest 1035 could be determined. For example, the combined region ofinterest 1035 could be determined as a polygon (e.g., such as arectangle) or an ellipse bounding all of the pixels in the first andsecond initial template regions 1010 and 1025. More generally, the unionof some or all initial template regions created from associated selectedreference points may be used to construct a bounding shape, such as apolygon, an ellipse, etc. Any point inside such a bounding shape is thenconsidered to be part of the created region of interest and, forexample, may serve as a brand identifier template.

Additionally or alternatively, a set of helper tools may be used tomodify, for example, the template region 1010 in a regular and precisemanner through subsequent input commands provided by the user 170. Forexample, instead of combining the initial template region 1010 with thesecond template region 1025 as described above, the user 170 can clickon a second point 1050 outside the shaded template region 1010 to causethe template region 1010 to grow to the selected second point 1050. Theresult is a new template region 1055. Similarly, the user 170 can clickon a third point (not shown) inside the shaded template region 1010 tocause the template region 1010 to shrink to the selected third point.

Furthermore, the user can access an additional set of helper tools tomodify the current template region (e.g., such as the template region1010) in more ways than only a straightforward shrinking or expanding ofthe template region to a selected point. In the illustrated example, thehelper tool used to modify the template region 1010 to become thetemplate region 1055 was a GROW_TO_POINT helper tool. Other examplehelper tools include a GROW_ONE_STEP helper tool, aGROW_ONE_DIRECTIONAL_STEP helper tool, a GROW_TO_POINT_DIRECTIONALhelper tool, an UNDO helper tool, etc. In the illustrated example,clicking on the selected point 1050 with the GROW_ONE_STEP helper toolactivated would cause the template region 1010 to grow by only one stepof resolution to become the new template region 1060. However, if theGROW_ONE_DIRECTIONAL_STEP helper tool were activated, the templateregion 1010 would grow by one step of resolution only in the directionof the selected point 1015 to become the new template region 1065 (whichcorresponds to the entire darker shaded region depicted in FIG. 10). Ifa GROW_TO_POINT_DIRECTIONAL helper tool were activated (example notshown), the template region 1010 would grow to the selected point, butonly in the direction of the selected point. In the case of theDIRECTIONAL helper tools, the helper tool determines the side, edge,etc., of the starting template region nearest the selected point todetermine in which direction the template region should grow.Additionally, other helper tools may be used to select the type, size,color, etc., of the shape/polygon (e.g., such as a rectangle) used tocreate the initial template region, to specify the resolution step size,etc. Also, although the example helper tools are labeled using the term“GROW” and the illustrated examples depict these helper tools asexpanding the template region 1010, these tools also can cause thetemplate region 1010 to shrink in a corresponding manner by selecting apoint inside, instead or outside, the example template region 1010. Assuch, the example helper tools described herein can cause a startingtemplate region to grow in either an expanding or contracting mannerdepending upon whether a point is selected outside or inside thetemplate region, respectively.

As mentioned above, the user 170 can also use the example automatedregion of interest creation technique to create a desired region ofinterest based on a stored and/or previously created region of interest(e.g., a reference region of interest) acting as a template. To create aregion of interest using a stored and/or previously created region ofinterest, the user 170 uses a mouse (or any other appropriate inputdevice) to select a reference point approximately in the center of thedesired region of interest. Alternatively, the user 170 could markmultiple reference points to define a boundary around the desired regionof interest. To indicate that the example GUI 152 should create theregion of interest from a stored and/or previously created region ofinterest rather than from scratch, the user 170 may use a differentmouse button (or input selector on the input device) and/or press apredetermined key while selecting the reference point(s), press a searchbutton on the graphical display before selecting the reference point(s),etc. After the reference point(s) are selected, the example GUI 152 usesany appropriate template matching procedure (e.g., such as thenormalized cross correlation template matching technique describedbelow) to match a region associated with the selected reference point(s)to one or more stored and/or previously created region of interest. TheGUI 152 then displays the stored and/or previously created region ofinterest that best matches the region associated with the selectedreference point(s). The user 170 may then accept the returned region ormodify the region using the helper tools as described above in thecontext of creating a region of interest from scratch.

In some cases, a user 170 may wish to exclude an occluded portion of adesired region of interest because, for example, some object ispositioned such that it partially obstructs the brand identifier(s)(e.g., logo(s)) included in the region of interest. For example, in thecontext of a media content presentation of a baseball game, a brandidentifier in a region of interest may be a sign or other advertisementposition behind home plate which is partially obstructed by the batter.In situations such as these, it may be more convenient to initiallyspecify a larger region of interest (e.g., the region corresponding tothe entire sign or other advertisement) and then exclude the occludedportion of the larger region (e.g., the portion corresponding to thebatter) to create the final, desired region of interest. To perform suchregion exclusion, the user 170 may use an EXCLUSION MARK-UP helper toolto create a new region that is overlaid (e.g., using a different color,shading, etc.) on a region of interest initially created from scratch orfrom a stored and/or previously created region of interest.Additionally, the helper tools already described above (e.g., such asthe GROW_TO_POINT, GROW_ONE_STEP, GROW_ONE_DIRECTIONAL_STEP, etc. helpertools) may be used to modify the size and/or shape of the overlaidregion. When the user 170 is satisfied with the overlaid region, the GUI152 excludes the overlaid region (e.g., corresponding to the occlusion)from the initially created region of interest to form the final, desiredregion of interest.

Returning to FIG. 2, once the information for the current scene,region(s) of interest, and/or brand identifiers included therein hasbeen provided via the GUI 152 for a new scene or a repeated scene ofchanged interest, the example GUI 152 updates the example learnedknowledge database 264 with information concerning, for example, thebrand identifier(s) (e.g., logo(s)), identity(ies), location(s),size(s), orientation(s), etc. The resulting updated information maysubsequently be used for comparison with another identified scenedetected in the media stream 160 of FIG. 1 and/or a scene included inany other media stream(s). Additionally, and as discussed above, atracker function is then initiated for each newly marked region ofinterest. The tracker function uses the marked region of interest as atemplate to track the corresponding region of interest in the adjacentimage frames comprising the current detected scene. In particular, anexample tracker function determines how a region of interest marked in akey frame of a scene may change (e.g., in location, size, orientation,etc.) over the adjacent image frames comprising the scene. Parametersdescribing the region of interest and how it changes (if at all) overthe scene are used to derive an exposure measurement for brandidentifier(s) included in the region of interest, as well as to updatethe example learned knowledge database 264 with information concerninghow the brand identifier(s) included in the region of interest mayappear in subsequent scenes.

Furthermore, if the marked region of interest contains an excludedregion representing an occluded portion of a larger region of interest(e.g., such as an excluded region marked using the EXCLUSION MARK-UPhelper tool described above), the example tracker function can beconfigured to track the excluded region separately to determine whetherthe occlusion represented by the excluded region changes and/or lessens(e.g., becomes partially or fully removed) in the adjacent frames. Forexample, the tracker function can use any appropriate image comparisontechnique to determine that at least portions of the marked region ofinterest and at least portions of the excluded region of interest havebecome similar to determine that the occlusion has changed and/orlessened. If the occlusion changes and/or lessens in the adjacentframes, the example tracker function can combine the marked region ofinterest with the non-occluded portion(s) of the excluded region toobtain a new composite region of interest and/or composite brandidentifier template (discussed below) for use in brand identifierrecognition. Next, to continue analysis of brand identifier exposure inthe media stream 160 of FIG. 1, the example scene recognizer 252restarts the media stream 160 of FIG. 1.

As discussed above, to recognize a brand identifier (e.g., logo)appearing in a scene, and to gather information regarding the brandidentifier, the example brand exposure monitor 150 of FIG. 2 includesthe brand recognizer 254 (also known as the logo detector 254). Theexample brand recognizer 254 of FIG. 2 determines all brand identifiersappearing in the scene. For example the brand recognizer 254 mayrecognize brand identifiers in a current scene of interest by comparingthe region(s) of interest with one or more reference brand identifiers(e.g., one or more reference logos) stored in the learned knowledgedatabase 264 and/or known brand identifiers stored in the brand library266. The reference brand identifier information may be stored using anydata structure(s) in the brand library 266 and/or the learned knowledgedatabase 264. For example, the brand library 266 and/or the learnedknowledge database 264 may store the reference brand identifierinformation using bitmap files, a repository of JPEG files, etc.

To reduce processing requirements and improve recognition efficiencysuch that, for example, brand identifiers may be recognized inreal-time, the example brand recognizer 254 uses known and/or learnedinformation to analyze the current scene for only those reference brandidentifiers expected to appear in the scene. For example, if the currentscene is a repeated scene of interest matching a reference (e.g.,previously learned or known) scene, the example brand recognizer 254 mayuse stored information regarding the matched reference scene todetermine the region(s) of interest and associated brand identifier(s)expected to appear in the current scene. Furthermore, the example brandrecognizer 254 may track the recognized brand identifiers appearing in ascene across the individual image frames comprising the scene todetermine additional brand identifier parameters and/or to determinecomposite brand identifier templates (as discussed above) to aid infuture recognition of brand identifiers, to provide more accurate brandexposure reporting, etc.

In an example implementation, the brand recognizer 254 performs templatematching to compare a region of interest in the current scene to one ormore reference brand identifiers (e.g., one or more reference logos)associated with the matching reference scene. For example, when a userinitially marks a brand identifier (e.g., logo) in a detected scene(e.g., such as a new scene), the marked region represents a region ofinterest. From this marked region of interest, templates of differentsizes, perspectives, etc. are created to be reference brand identifiersfor the resulting reference scene. Additionally, composite referencebrand identifier templates may be formed by the example tracker functiondiscussed above from adjacent frames containing an excluded region ofinterest representing an occlusion that changes and/or lessens. Then,for a new detected scene, template matching is performed against thesevarious expected reference brand identifier(s) associated with thematching reference scene to account for possible (and expected)perspective differences (e.g., differences in camera angle, zooming,etc.) between a reference brand identifier and its actual appearance inthe current detected scene. For example, a particular reference brandidentifier may be scaled from one-half to twice its size, inpredetermined increments, prior to template matching with the region ofinterest in the current detected scene. Additionally or alternatively,the orientation of the particular reference brand identifier may bevaried over, for example, −30 degrees to +30 degrees, in predeterminedincrements, prior to template matching with the region of interest inthe current detected scene. Furthermore, template matching asimplemented by the example brand recognizer 254 may be based oncomparing the luminance values, chrominance values, or any combinationthereof, for the region(s) of interest and the reference brandidentifiers.

An example template matching technique that may be implemented by theexample brand recognizer 254 for comparing a region of interest to thescaled versions and/or different orientations of the reference brandidentifiers is described in the paper “Fast Normalized CrossCorrelation” by J. P. Lewis, available athttp://www.idiom.com/˜zilla/Work/nvisionInterface/nip.pdf (accessed Oct.24, 2007), which is submitted herewith and incorporated by reference inits entirety. In an example implementation based on the templatematching technique described by Lewis, the example brand recognizer 254computes the normalized cross correlation (e.g., based on luminanceand/or chrominance values) of the region of interest with each templaterepresentative of a particular reference brand identifier having aparticular scaling and orientation. The largest normalized crosscorrelation across all templates representative of all the differentscalings and orientations of all the different reference brandidentifiers of interest is then associated with a match, provided thecorrelation exceeds a threshold. As discussed in Lewis, the benefits ofa normalized cross correlation implementation include robustness tovariations in amplitude of the region of interest, robustness to noise,etc. Furthermore, such an example implementation of the example brandrecognizer 254 can be implemented using Fourier transforms and runningsums as described in Lewis to reduce processing requirements over abrute-force spatial domain implementation of the normalized crosscorrelation.

To report measurements and other information about brand identifiers(e.g., logos) recognized and/or detected in the example media stream160, the example brand exposure monitor 150 of FIG. 2 includes a reportgenerator 256. The example report generator 256 of FIG. 2 collects thebrand identifiers, along with any associated appearance parameters,etc., determined by the brand recognizer 254, organizes the information,and produces a report. The report may be output using any technique(s)such as, for example, printing to a paper source, creating and/orupdating a computer file, updating a database, generating a display,sending an email, etc.

While an example manner of implementing the example brand exposuremonitor 150 of FIG. 1 has been illustrated in FIG. 2, some or all of theelements, processes and/or devices illustrated in FIG. 2 may becombined, divided, re-arranged, omitted, eliminated and/or implementedin any way. Further, the example scene recognizer 252, the example brandrecognizer 254, the example GUI 152, the example mass memory 260, theexample scene database 262, the example learned knowledge database 264,the example brand library 266, the report generator 256, and/or moregenerally, the example brand exposure monitor 150 may be implemented byhardware, software, firmware and/or any combination of hardware,software and/or firmware. Thus, for example, any of the example scenerecognizer 252, the example brand recognizer 254, the example GUI 152,the example mass memory 260, the example scene database 262, the examplelearned knowledge database 264, the example brand library 266, thereport generator 256, and/or more generally, the example brand exposuremonitor 150 could be implemented by one or more circuit(s), programmableprocessor(s), application specific integrated circuit(s) (ASIC(s)),programmable logic device(s) (PLD(s)) and/or field programmable logicdevice(s) (FPLD(s)), etc. When any of the appended claims are read tocover a purely software implementation, at least one of the examplebrand exposure monitor 150, the example scene recognizer 252, theexample brand recognizer 254, the example GUI 152, the example massmemory 260, the example scene database 262, the example learnedknowledge database 264, the example brand library 266 and/or the reportgenerator 256 are hereby expressly defined to include a tangible mediumsuch as a memory, digital versatile disk (DVD), compact disk (CD,) etc.Moreover, the example brand exposure monitor 150 may include one or moreelements, processes, and/or devices instead of, or in addition to, thoseillustrated in FIG. 2, and/or may include more than one of any or all ofthe illustrated, processes and/or devices.

An example manner of implementing the example scene recognizer 252 ofFIG. 2 is shown in FIG. 3. The example scene recognizer 252 of FIG. 3includes a signature generator 352 to create one or more imagesignatures for each frame (possibly after sub-sampling) included in, forexample, the media stream 160. The one or more image signatures are thenused for scene identification and/or scene change detection. In theillustrated example, the signature generator 252 generates the imagesignature for an image frame included in the media stream 160 bycreating an image histogram of the luminance and/or chrominance valuesincluded in the image frame.

To implement scene identification and scene change detection asdiscussed above, the example scene recognizer 252 of FIG. 3 includes ascene detector 354. The example scene detector 354 of the illustratedexample detects scenes in the media stream 160 by comparing successiveimage frames to a starting frame representative of a current scene beingdetected. As discussed above, successive image frames that have similarimage signatures are grouped together to form a scene. One or moreimages and their associated signatures are then used to form the keyframe(s) and associated key signature(s) for the detected frame. In theillustrated example, to detect a scene by determining whether a scenechange has occurred, the example scene detector 354 compares thegenerated image signature for a current image frame with the imagesignature for the starting frame (or the appropriate key frame) of thescene currently being detected. If the generated image signature for thecurrent image frame is sufficiently similar to the starting frame's (orkey frame's) image signature (e.g., when negligible motion has occurredbetween successive frames in the media stream 160, when the cameraparameters are substantially the same and the backgrounds arestatistically stationary, etc.), the example scene detector 354 includesthe current image frame in the current detected scene, and the nextframe is then analyzed by comparing it to the starting frame (or keyframe) of the scene. However, if the scene detector 354 detects asignificant change between the image signatures (e.g., in the example ofpresentation of a baseball game, when a batter in the preceding frame isreplaced by an outfielder in the current image frame of the media stream160), the example scene detector 354 identifies the current image frameas starting a new scene, stores the current image frame as the startingframe (and/or key frame) for that new scene, and stores the imagesignature for the current frame for use as the starting image signature(and/or key signature) for that new scene. The example scene detectorthen marks the immediately previous frame as the ending frame for thecurrent scene and determines one or more key frames and associated keyimage signature(s) to represent the current scene. As discussed above,the key frame and key image signature for the current scene may bechosen to be, for example, the frame and signature corresponding to thefirst frame in the scene, the last frame in the scene, the midpointframe in the scene, etc. In another example, the key frame and key imagesignature may be determined to be an average and/or some otherstatistical combination of the frames and/or signatures corresponding tothe detected scene. The current scene is then ready for sceneclassification.

The example scene recognizer 252 of FIG. 3 also includes a sceneexcluder 355 to exclude certain detected scenes from brand exposureprocessing. As discussed above, a detected scene may be excluded undercircumstances where it is likely the scene will not contain any brandidentifiers (e.g., logos). In an example implementation, the sceneexcluder 355 is configured to use domain knowledge corresponding to theparticular type of media content being processed to determine when adetected scene exhibits characteristics indicating that the scene willnot contain any brand identifiers. If a detected scene is excluded, thescene excluder 355 of the illustrated example invokes the reportgenerator 256 of FIG. 2 to report the excluded scene.

To categorize a non-excluded, detected scene, the example scenerecognizer 252 of FIG. 2 includes a scene classifier 356. The examplescene classifier 356 compares the current detected scene (referred to asthe current scene) to one or more reference scenes (e.g., previouslylearned and/or known scenes) stored in the scene database 262 and/or thelearned knowledge database 264. For example, the scene classifier 356may compare an image signature representative of the current scene toone or more reference image signatures representative of one or morerespective reference scenes. Based on the results of the comparison, theexample scene classifier 356 classifies the current scene into one ofthe following four categories: a repeated scene of interest, a repeatedscene of changed interest, a new scene, or a scene of no interest. Forexample, if the current scene's image signature does not match anyreference scene's image signature, the example scene classifier 356classifies the scene as a new scene and displays the current scene viathe output device 270 on FIG. 2. For example, a prompt is shown via theGUI 152 to alert the user 170 of the need to identify the scene.

However, if the current scene's image signature matches a referencescene's image signature and the corresponding reference scene hasalready been marked as a scene of no interest, information describingthe current scene (e.g., such as its key frame, key signature, etc.) foruse in detecting subsequent scenes of no interest is stored in theexample learned knowledge database 264 and the next scene in the mediastream 160 is analyzed. If, however, a match is found and thecorresponding reference scene has not been marked as a scene of nointerest, the example scene classifier 356 then determines one or moreexpected regions of interest in the detected scene based on region ofinterest information corresponding to the matched reference scene. Theexample scene classifier 356 then invokes the example brand recognizer254 to perform brand recognition by comparing the expected region(s) ofinterest included in the current scene with one or more reference brandidentifiers (e.g., previously learned and/or known brand identifiers)stored in the learned knowledge database 264 and/or the brand library266. Then, if one or more regions of interest included in the identifiedscene do not match any of the corresponding expected reference brandidentifier stored in the learned knowledge database 264 and/or the branddatabase 266, the current scene is classified as a repeated scene ofchanged interest and displayed at the output device 270. For example, aprompt may be shown via the GUI 152 to alert the user 170 of the need todetect and/or identify one or more brand identifiers included in thenon-matching region(s) of interest. The brand identifier (e.g., logo)marking/identification provided by the user 170 is then used to updatethe learned knowledge database 264. Additionally, if the current scenewas a new scene, the learned knowledge database 264 may be updated touse the current detected scene as a reference for detecting futurerepeated scenes of interest. However, if all of the region(s) ofinterest included in the current scene match the corresponding expectedreference brand identifier(s) stored in the learned knowledge database264 and/or the brand database 266, the current scene is classified as arepeated scene of interest and the expected region(s) of interest areautomatically analyzed by the brand recognizer 254 to provide brandexposure reporting with no additional involvement needed by the user170. Furthermore, and as discussed above, a tracker function is theninitiated for each expected region of interest. The tracker functionuses the expected region of interest as a template to track thecorresponding region of interest in the adjacent image frames comprisingthe current detected scene. Parameters describing the region of interestand how it changes over the scene are used to derive an exposuremeasurement for brand identifier(s) included in the region of interest,as well as to update the example learned knowledge database 264 withinformation concerning how the brand identifier(s) included in theregion of interest may appear in subsequent scenes.

While an example manner of implementing the example scene recognizer 252of FIG. 2 has been illustrated in FIG. 3, some or all of the elements,processes and/or devices illustrated in FIG. 3 may be combined, divided,re-arranged, omitted, eliminated and/or implemented in any way. Further,the example signature generator 352, the example scene detector 354, theexample scene classifier 356, and/or more generally, the example scenerecognizer 252 of FIG. 3 may be implemented by hardware, software,firmware and/or any combination of hardware, software and/or firmware.Moreover, the example scene recognizer 252 may include data structures,elements, processes, and/or devices instead of, or in addition to, thoseillustrated in FIG. 3 and/or may include more than one of any or all ofthe illustrated data structures, elements, processes and/or devices.

An example manner of implementing the example brand recognizer 254 ofFIG. 2 is shown in FIG. 4. To detect one or more brand identifiers(e.g., one or more logos) in a region of interest of the identifiedscene, the example brand recognizer 254 of FIG. 4 includes a brandidentifier detector 452. The example brand identifier detector 452 ofFIG. 4 compares the content of each region of interest specified by, forexample, the scene recognizer 252 of FIG. 2, with one or more reference(e.g., previously learned and/or known) brand identifiers correspondingto a matched reference scene and stored in the example learned knowledgedatabase 264 and/or the brand library 266. For example, and as discussedabove, the brand identifier detector 452 may perform template matchingto compare the region of interest to one or more scaled versions and/orone or more different orientations of the reference brand identifiers(e.g., reference logos) stored in the learned knowledge database 264and/or the brand library 266 to determine brand identifiers (e.g.,logos) included in the region of interest.

For example, the brand identifier detector 452 may include a region ofinterest (ROI) detector 464 and a ROI tracker 464. The example ROIdetector 462 locates an ROI in a key frame representing the currentscene by searching each known or previously learned (e.g., observed) ROIassociated with the current scene's matching reference scene.Additionally, the example ROI detector 462 may search all known orpreviously learned locations, perspectives (e.g., size, angle, etc.),etc., for each expected ROI associated with the matching referencescene. Upon finding an ROI in the current scene that matches an expectedROI in the reference scene, the observed ROI, its locations, itsperspectives, and its association with the current scene are stored inthe learned knowledge database 264. The learned knowledge database 264,therefore, is updated with any new learned information each time an ROIis detected in a scene. The example ROI tracker 464 then tracks the ROIdetected by the ROI detector 464 in the key frame of the current scene.For example, the ROI tracker 464 may search for the detected ROI inimage frames adjacent to the key frame of the current scene, and in aneighborhood of the known location and perspective of the detected ROIin the key frame of the current scene. During the tracking process,appearance parameters, such as, for example, location, size, matchingquality, visual quality, etc. are recorded for assisting the detectionof ROI(s) in future repeated image frames and for deriving exposuremeasurements. (These parameters may be used as search keys and/ortemplates in subsequent matching efforts.) The example ROI tracker 464stops processing the current scene when all frames in the scene areprocessed and/or when the ROI cannot be located in a certain specifiednumber of consecutive image frames.

To identify the actual brands associated with one or more brandidentifiers, the example brand recognizer 254 of FIG. 4 includes a brandidentifier matcher 454. The example brand identifier matcher 454processes the brand identifier(s) detected by the brand identifierdetector 452 to obtain brand identity information stored in the branddatabase 266. For example, the brand identity information stored in thebrand database 266 may include, but is not limited to, internalidentifiers, names of entities (e.g., corporations, individuals, etc.)owning the brands associated with the brand identifiers, product names,service names, etc.

To measure the exposure of the brand identifiers (e.g., logos) detectedin, for example, the scenes detected in the media stream 160, theexample brand recognizer 254 of FIG. 4 includes a measure and trackingmodule 456. The example measure and tracking module 456 of FIG. 4collects appearance data corresponding to the detected/recognized brandidentifier(s) (e.g., logos) included in the image frames of eachdetected scene, as well as how the detected/recognized brandidentifier(s) (e.g., logos) may vary across the image frames comprisingthe detected scene. For example, such reported data may includeinformation regarding location, size, orientation, match quality, visualquality, etc., for each frame in the detected scene. (This data enablesa new ad payment/selling model wherein advertisers pay per frame and/ortime of exposure of embedded brand identifiers.) In an exampleimplementation, the measure and tracking module 456 determines aweighted location and size for each detected/recognized brandidentifier. For example, the measure and tracking module 456 may weightthe location and/or size of a brand identifier by the duration ofexposure at that particular location and/or size to determine theweighted location and/or size information. A report of brand exposuremay be generated from the aforementioned information by a reportgenerator 256.

While an example manner of implementing the example brand recognizer 254of FIG. 2 has been illustrated in FIG. 4, some or all of the elements,processes and/or devices illustrated in FIG. 4 may be combined, divided,re-arranged, omitted, eliminated and/or implemented in any way. Further,the example brand identifier detector 452, the example brand identifiermatcher 454, the example measure and tracking module 456, and/or moregenerally, the example brand recognizer 254 of FIG. 4 may be implementedby hardware, software, firmware and/or any combination of hardware,software and/or firmware. Thus, for example, any of the example brandidentifier detector 452, the example brand identifier matcher 454, theexample measure and tracking module 456, and/or more generally, theexample brand recognizer 254 could be implemented by one or morecircuit(s), programmable processor(s), application specific integratedcircuit(s) (ASIC(s)), programmable logic device(s) (PLD(s)) and/or fieldprogrammable logic device(s) (FPLD(s)), etc. When any of the appendedclaims are read to cover a purely software implementation, at least oneof the example brand recognizer 254, the example brand identifierdetector 452, the example brand identifier matcher 454 and/or theexample measure and tracking module 456 are hereby expressly defined toinclude a tangible medium such as a memory, digital versatile disk(DVD), compact disk (CD,) etc. Moreover, the example brand recognizer254 of FIG. 4 may include one or more elements, processes, and/ordevices instead of, or in addition to, those illustrated in FIG. 4,and/or may include more than one of any or all of the illustratedelements, processes and/or devices.

To better illustrate the operation of the example signature generator352, the example scene detector 354, the example scene excluder 355, theexample scene classifier 356, the example ROI detector 464 and theexample ROI tracker 464, example scenes that could be processed tomeasure brand exposure are shown in FIGS. 5A-5D. The example scenesshown in FIGS. 5A-5D are derived from a media stream 160 providingexample broadcasts of sporting events. The example sporting eventbroadcasts are merely illustrative and the methods and apparatusdisclosed herein are readily applicable to processing media streams todetermine brand exposure associated with any type of media content.Turning to the figures, FIG. 5A illustrates four example key frames 505,510, 515 and 525 associated with four respective example scenes whichcould qualify for scene exclusion based on known or learned domainknowledge. In particular, the example key frame 505 depicts a scenehaving a background (e.g., a golf course) that includes a predominanceof uniformly grouped greenish pixels. If the domain knowledgecorresponding to the type of media content that generated the examplekey frame 505 (e.g., such as an expected broadcast of a golfing event)indicates that a scene including a predominance of uniformly groupedgreenish pixels should be excluded because it corresponds to a camerashot of the playing field (e.g., golf course), the example sceneexcluder 355 could be configured with such knowledge and exclude thescene corresponding to the example key frame 505.

The example key frame 510 corresponds to a scene of short durationbecause the scene includes a predominance of components in rapid motion.In an example implementation, the scene excluder 355 could be configuredto exclude such a scene because it is unlikely a brand identifier wouldremain in the scene for sufficient temporal duration to be observedmeaningfully by a person consuming the media content. The example keyframe 515 corresponds to a scene of a crowd of spectators at thebroadcast sporting event. Such a scene could also be excluded by theexample scene excluder 355 if, for example, the domain knowledgeavailable to the scene excluder 355 indicated that a substantiallyuniform, mottled scene corresponds to an audience shot and, therefore,is not expected to include any brand identifier(s). The example keyframe 520 corresponds to a scene from a commercial being broadcastduring the example broadcasted sporting event. In an exampleimplementation, the scene excluder 355 could be configured to excludesscenes corresponding to a broadcast commercial (e.g., based on adetected audio code in the example media stream 160, based on a detectedtransition (e.g., blank screen) in the example media stream 160, etc.)if, for example, brand exposure reporting is to be limited to embeddedadvertisements.

FIG. 5B illustrates two example key frames 525 and 530 associated withtwo example scenes which could be classified as scenes of no interest bythe example scene classifier 356. In the illustrated example, the keyframe 525 corresponds to a new detected scene that may be marked by theuser 170 as a scene of no interest because the example key frame 525does not include any brand identifiers (e.g., logos). The scenecorresponding to the key frame 525 then becomes a learned referencescene of no interest. Next, the example key frame 530 corresponds to asubsequent scene detected by the example scene detector 354. Bycomparing the similar image signatures (e.g., image histograms) for thekey frame 520 to the key frame 530 of the subsequent detected scene, theexample scene classifier 356 may determine that the image frame 530corresponds to a repeat of the reference scene corresponding to theexample key frame 525. In such a case, the example scene classifier 356would then determine that the key frame 530 corresponds to a repeatedscene of no interest because the matching reference scene correspondingto the example key frame 525 had been marked as a scene of no interest.

FIG. 5C illustrates two example key frames 535 and 540 associated withtwo example scenes which could be classified as scenes of interest bythe example scene classifier 356. In the illustrated example, the keyframe 535 corresponds to a new detected scene that may be marked by theuser 170 as a scene of interest because the example key frame 535 has aregion of interest 545 including a brand identifier (e.g., the signadvertising “Banner One”). The scene corresponding to the key frame 535would then become a learned reference scene of interest. Furthermore,the user 170 may mark the region of interest 545, which would then beused by the example ROI tracker 464 to create one or more referencebrand identifier templates for detecting subsequent repeated scenes ofinterest corresponding to this reference scene and region of interest.Next, the example key frame 540 corresponds to a subsequent scenedetected by the example scene detector 354. By comparing the similarimage signatures (e.g., image histograms) for the key frame 535 and thekey frame 540 of the subsequent detected scene, the example sceneclassifier 356 may determine that the image frame 540 corresponds to arepeat of the reference scene corresponding to the example key frame535. Because the reference scene is a scene of interest, the examplescene classifier 356 would then invoke the example ROI detector 462 tofind the appropriate expected region of interest in the example keyframe 540 based on the reference template(s) corresponding to thereference region of interest 535. In the illustrated example, the ROIdetector 462 finds the region of interest 550 corresponding to thereference region of interest 535 because the two regions of interest aresubstantially similar except for expected changes in orientation, size,location, etc. Because the example ROI detector 462 found and verifiedthe expected region of interest 550 in the illustrated example, theexample scene classifier 356 would classify the scene corresponding tothe example key frame 540 as a repeated scene of interest relative tothe reference scene corresponding to the example key frame 535.

FIG. 5D illustrates two example key frames 555 and 560 associated withtwo example scenes which could be classified as scenes of interest bythe example scene classifier 356. In the illustrated example, the keyframe 555 corresponds to a new detected scene that may be marked by theuser 170 as a scene of interest because the example key frame 555 has aregion of interest 565 including a brand identifier (e.g., the signadvertising “Banner One”). The scene corresponding to the key frame 555would then become a learned reference scene of interest. Furthermore,the user 170 may mark the region of interest 565, which would then beused by the example ROI tracker 464 to create one or more referencebrand identifier templates for detecting subsequent repeated scenes ofinterest corresponding to this reference scene and region of interest.Next, the example key frame 560 corresponds to a subsequent scenedetected by the example scene detector 354. By comparing the similarimage signatures (e.g., image histograms) for the key frame 555 and thekey frame 560 of the subsequent detected scene, the example sceneclassifier 356 may determine that the image frame 560 corresponds to arepeat of the reference scene corresponding to the example key frame555. Because the reference scene is a scene of interest, the examplescene classifier 356 would then invoke the example ROI detector 462 tofind the appropriate expected region of interest in the example keyframe 560 based on the reference template(s) corresponding to thereference region of interest 565.

In the illustrated example, the ROI detector 462 does not find anyregion of interest corresponding to the reference region of interest 565because there is no brand identifier corresponding to the advertisement“Banner One” in the example key frame 560. Because the example ROIdetector was unable to verify the expected region of interest in theillustrated example, the example scene classifier 356 would classify thescene corresponding to the example key frame 560 as a repeated scene ofchanged interest relative to the reference scene corresponding to theexample key frame 555. Next, because the scene corresponding to theexample key frame 560 is classified as a repeated scene of changedinterest, the user 170 would be requested to mark any brandidentifier(s) included in the scene. In the illustrated example, theuser 170 may mark the region of interest 570 because it includes a brandidentifier corresponding to a sign advertising “Logo Two.” The exampleROI tracker 464 would then be invoked to create one or more referencebrand identifier templates based on the marked region of interest 570for detecting subsequent repeated scenes of interest including this newreference region of interest.

FIGS. 6A-6 B collectively form a flowchart representative of examplemachine accessible instructions 600 that may be executed to implementthe example scene recognizer 252 of FIGS. 2 and/or 3, and/or at least aportion of the example brand exposure monitor 150 of FIGS. 1 and/or 2.FIGS. 7A-7C collectively form a flowchart representative of examplemachine accessible instructions 700 that may be executed to implementthe example GUI 152 of FIGS. 1 and/or 2, and/or at least a portion ofthe example brand exposure monitor 150 of FIGS. 1 and/or 2. FIGS. 8A-8Bare flowcharts representative of example machine accessible instructions800 and 850 that may be executed to implement the example brandrecognizer 254 of FIGS. 2 and/or 3, and/or at least a portion of theexample brand exposure monitor 150 of FIGS. 1 and/or 2. The examplemachine accessible instructions of FIGS. 6A-6B, 7A-7C and/or 8A-8B maybe carried out by a processor, a controller and/or any other suitableprocessing device. For example, the example machine accessibleinstructions of FIGS. 6A-6B, 7A-7C and/or 8A-8B may be embodied in codedinstructions stored on a tangible medium such as a flash memory, aread-only memory (ROM) and/or random-access memory (RAM) associated witha processor (e.g., the example processor 905 discussed below inconnection with FIG. 9). Alternatively, some or all of the example brandexposure monitor 150, the example GUI 152, the example scene recognizer252, and/or the example brand recognizer 254 may be implemented usingany combination(s) of application specific integrated circuit(s)(ASIC(s)), programmable logic device(s) (PLD(s)), field programmablelogic device(s) (FPLD(s)), discrete logic, hardware, firmware, etc.Also, some or all of the example machine accessible instructions ofFIGS. 6A-B, 7A-C and/or 8A-8B may be implemented manually or as anycombination of any of the foregoing techniques, for example, anycombination of firmware, software, discrete logic and/or hardware.Further, although the example machine accessible instructions aredescribed with reference to the example flowcharts of FIGS. 6A-6B, 7A-7Cand 8A-8B, many other methods of implementing the machine accessibleinstructions of FIGS. 6A-6B, 7A-7C, and/or 8A-8B may be employed. Forexample, the order of execution of the blocks may be changed, and/or oneor more of the blocks described may be changed, eliminated, sub-divided,or combined. Additionally, some or all of the example machine accessibleinstructions of FIGS. 6A-6, 7A-7C and/or 8A-8B may be carried outsequentially and/or carried out in parallel by, for example, separateprocessing threads, processors, devices, discrete logic, circuits, etc.

Turning to FIGS. 6A-6B, execution of the example machine executableinstructions 600 begins with the example scene recognizer 252 includedin the example brand exposure monitor 150 receiving a media stream, suchas the example media stream 160 of FIG. 1 (block 602 of FIG. 6A). Theexample scene recognizer 252 then detects a scene included in thereceived media stream 160 by comparing image signatures created forimage frames of the media stream 160 (block 604). As discussed above,successive image frames that have substantially similar image signatures(e.g., such as substantially similar image histograms) are identified tobe part of the same scene. In an example implementation, one of thesubstantially similar image frames will be stored as a key framerepresentative of the scene, and the image signature created for the keyframe will serve as the detected scene's image signature. An exampletechnique for generating the image signature at block 604 which usesimage histograms is discussed above in connection with FIG. 2.

Next, the example scene recognizer 252 performs scene exclusion byexamining the key frame of the current scene detected at block 604 forcharacteristics indicating that the scene does not include any brandidentifiers (e.g., logos) (block 606). For example, and as discussedabove, domain knowledge specific to the type of media content expectedto be processed may be used to configure the example scene recognizer252 to recognize scene characteristics indicative of a scene lacking anybrand identifiers that could provide brand exposure. In the context ofthe baseball game example, a scene characterized as primarily includinga view of a blue sky (e.g., when following pop-up fly ball), a view ofthe ground (e.g., when following a ground ball), or having a quicklychanging field of view (e.g., such as when a camera pans to follow abase runner), etc., may be excluded at block 606.

If the current detected scene is excluded (block 608), the example scenerecognizer 252 invokes the report generator 256 to report the exclusionof the current detected scene (block 610). Additionally oralternatively, the example scene recognizer 252 may store informationdescribing the excluded scene as learned knowledge to be used to excludefuture detected scenes and/or classify future scenes as scenes of nointerest. The example scene recognizer 252 then examines the mediastream 160 to determine whether the media stream 160 has ended (block630). If the end of the media stream 160 has been reached, execution ofthe example machine accessible instructions 600 ends. If the mediastream 160 has not completed (block 630), control returns to block 604to allow the example scene recognizer 252 to detect a next scene in themedia stream 160.

Returning to block 608, if the current detected scene (also referred toas the “current scene”) is not excluded, the example scene recognizer252 compares the current scene with one or more reference (e.g.,previously learned and/or known) scenes stored in one or more databases(e.g., the scene database 262 and/or the learned knowledge database 264of FIG. 2) (block 612). An example technique for performing thecomparison at block 612 is discussed above in connection with FIG. 2.For example, at block 612 the example scene recognizer 252 may comparethe image signature (e.g., image histogram) for the current scene withthe image signatures (e.g., image histograms) for the reference scenes.A signature match may be declared if the current scene's signature has acertain degree of similarity with a reference scene's signature asspecified by one or more thresholds. Control then proceeds to block 614of FIG. 6B.

If the example scene recognizer 252 determines that the image signatureof the current scene does not match any reference (e.g., previouslylearned and/or known) scene's signature (block 614), the scene isclassified as a new scene (block 626). The example scene recognizer 252then stops (e.g., pauses) the media stream 160 and passes the scenealong with the scene classification information to the example GUI 152to enable identification of the scene and any brand identifier(s) (e.g.,logos) included in the scene (block 627). Example machine readableinstructions 700 that may be executed to perform the identificationprocedure at block 627 are illustrated in FIGS. 7A-7C and discussed ingreater detail below. After any identification via the GUI 152 isperformed at block 627, the example scene recognizer 252 restarts themedia stream 160 and control proceeds to block 610 of FIG. 6A at whichthe example scene recognizer 252 invokes the report generator 256 toreport brand exposure based on the identification of the current sceneand/or brand identifier(s) included therein obtained at block 627.Control then returns to block 630 to determine whether there are morescenes remaining in the media stream 160.

Returning to block 614 of FIG. 6B, if the image signature of the currentscene matches an image signature corresponding to a reference (e.g.,previously learned and/or known) scene, a record of stored informationassociated with the matched reference scene is retrieved (block 616). Ifthe matched reference scene was marked and/or was otherwise determinedto be a scene of no interest (block 618) (e.g., a scene known to notinclude brand identifiers), the current scene is classified as a sceneof no interest (block 619). Control proceeds to block 610 of FIG. 6A atwhich the example scene recognizer 252 invokes the report generator 256to report that the current scene has been classified as a scene of nointerest. Control then returns to block 630 to determine whether thereare more scenes remaining in the media stream 160.

However, if the reference scene was not marked or otherwise determinedto be a scene of no interest (block 618), one or more regions ofinterest are then determined for the current scene (block 620). Theregion(s) of interest are determined based on stored region of interestinformation obtained at block 616 for the matched reference scene. Thedetermined region(s) of interest in the scene is(are) then provided tothe example brand recognizer 254 to enable comparison with one or morereference (e.g., previously learned and/or known) brand identifiers(block 621). Example machine readable instructions 800 that may beexecuted to perform the comparison procedure at block 621 areillustrated in FIG. 8A and discussed in greater detail below.

Based on the processing at block 621 performed by, for example, theexample brand recognizer 254, if the example scene recognizer 252determines that at least one region of interest does not match anyreference (previously learned and/or known) brand identifiers (block622), the scene is classified as a repeated scene of changed interest(block 628). A region of interest in a current scene may not match anyreference brand identifier(s) associated with the matched referencescene if, for example, the region of interest includes brandidentifier(s) (e.g., logos) that are animated, virtual and/or changingover time, etc. The example scene recognizer 252 then stops (e.g.,pauses) the media stream 160 and provides the scene, the sceneclassification, and the region(s) of interest information to the exampleGUI 152 to enable identification of the scene and any brandidentifier(s) included in the scene (block 629). Example machinereadable instructions 700 that may be executed to perform theidentification procedure at block 629 are illustrated in FIGS. 7A-7C anddiscussed in greater detail below. After any identification via the GUI152 is performed at block 629, the example scene recognizer restarts themedia stream 160 and control proceeds to block 610 of FIG. 6A at whichthe example scene recognizer 252 invokes the report generator 256 toreport brand exposure based on the identification of the current sceneand/or brand identifier(s) included therein obtained at block 629.Control then returns to block 630 to determine whether there are morescenes remaining in the media stream 160.

Returning to block 622, if all regions of interest in the scene matchreference (e.g., previously learned and/or known) brand identifiers, theexample scene recognizer 252 classifies the scene as a repeated scene ofinterest (block 624). The example scene recognizer 252 then provides thescene, the determined region(s) of interest and the detected/recognizedbrand identifier(s) to, for example, the example brand recognizer 254 toenable updating of brand identifier characteristics, and/or collectionand/or calculation of brand exposure information related to thedetected/recognized the brand identifier(s) (block 625). Example machinereadable instructions 850 that may be executed to perform the processingat block 625 are illustrated in FIG. 8B and discussed in greater detailbelow. Next, control proceeds to block 610 of FIG. 6A at which theexample scene recognizer 252 invokes the report generator 256 to reportbrand exposure based on the brand identifier(s) recognized/detected atblock 625. Control then returns to block 630 to determine whether thereare more scenes remaining in the media stream 160.

Turning to FIGS. 7A-7C, execution of the machine executable instructions700 begins with the GUI 152 receiving a detected scene and aclassification for the scene from, for example, the example scenerecognizer 252 or via processing performed at block 627 and/or block 629of FIG. 6B (block 701). The example GUI 152 then displays the scene via,for example, the output device 270 (block 702). The example GUI 152 thenevaluates the scene classification received at block 701 (block 704). Ifthe scene is classified as a new scene (block 706), the example GUI 152then prompts the user 170 to indicate whether the current scene is ascene of interest (or, in other words, is not a scene of no interest)(block 708). In the illustrated example, the current scene will defaultto be a scene of no interest unless the user indicates otherwise. Forexample, at block 708 the GUI 152 may prompt the user 170 to enteridentifying information, a command, click a button, etc., to indicatewhether the scene is of interest or of no interest. Additionally oralternatively, the GUI 152 may automatically determine that the scene isof no interest if the user 170 does not begin to mark one or moreregions of interest in the current scene within a predetermined intervalof time after the scene is displayed. If the user 170 indicates that thescene is of no interest (e.g., by affirmative indication or by failingto enter any indication regarding the current scene) (block 710), thedetected scene is reported to be a scene of no interest (block 712) andexecution of the example machine accessible instructions 700 then ends.However, if the user 170 indicates that the scene is of interest (block710), the user 170 may input a scene title for the current scene (block714). The example GUI 152 then stores the scene title (along with theimage signature) for the current scene in a database, (e.g., such as thelearned knowledge database 264) (block 716). After processing at block716 completes, or if the scene was not categorized as a new scene (block706), control proceeds to block 718 of FIG. 7B.

Next, the example GUI 152 prompts the user 170 to click on a region ofinterest in the displayed scene (block 718). Once the user 170 hasclicked on the region of interest, the example GUI 152 determines atwhich point the user 170 clicked and determines a small region aroundthe point clicked (block 720). The example GUI 152 then calculates theregion of interest and highlights the region of interest in the currentscene being displayed via the output 270 (block 722). If the user 170then clicks an area inside or outside of the highlighted displayedregion of interest to resize and/or reshape the region of interest(block 724), the example GUI 152 re-calculates and displays the updatedregion of interest. Control returns to block 724 to allow the user 170to continue re-sizing or re-shaping the highlighted, displayed region ofinterest. In another implementation, the region of interest creationtechnique of blocks 718-726 can be adapted to implement the exampleautomated region of interest creation technique described above inconnection with FIG. 10.

If the GUI 152 detects that the user 170 has not clicked an area insideor outside the highlighted region within a specified period of time(block 724), the example GUI 152 then compares the region of interestcreated by the user 170 with one or more reference (e.g., previouslylearned and/or known) brand identifiers (block 728). For example, atblock 728 the example GUI 152 may provide the created region of interestand current scene's classification of, for example, a new scene or arepeated scene of changed interest to the example brand recognizer 254to enable comparison with one or more reference (e.g., previouslylearned and/or known) brand identifiers. Additionally, if the scene isclassified as a new scene, as opposed to a repeated scene of changedinterest, the example brand recognizer 254 may relax the comparisonparameters to return brand identifiers that are similar to, but that donot necessarily match, the created region of interest. Example machinereadable instructions 800 that may be executed to perform the comparisonprocedure at block 728 are illustrated in FIG. 8A and discussed ingreater detail below.

Next, after the brand identifier(s) is(are) compared at block 728, theexample GUI 152 displays the closest matching reference (e.g.,previously learned and/or known) brand identifier to the region ofinterest (block 730). The example GUI 152 then prompts the user toaccept the displayed brand identifier or to input a new brand identifierfor the created region of interest (block 732). Once the user hasaccepted the brand identifier displayed by the example GUI 152 and/orhas input a new brand identifier, the example GUI 152 stores thedescription of the region of interest and the brand identifier in adatabase (e.g., such as the learned knowledge database 264) (block 734).For example, the description of the region of interest and/or brandidentifier(s) contained therein may include, but is not limited to,information related to the size, shape, color, location, texture,duration of exposure, etc. Additionally or alternatively, the exampleGUI 152 may provide the information regarding the created region(s) ofinterest and the identified brand identifier(s) to, for example, theexample brand recognizer 254 to enable reporting of the brandidentifier(s). Example machine readable instructions 850 that may beexecuted to perform the processing at block 734 are illustrated in FIG.8B and discussed in greater detail below.

Next, if the user 170 indicates that there are more regions of interestto be identified in the current scene (e.g., in response to a prompt)(block 736), control returns to block 718 at which the GUI 152 promptsthe user to click on a new region of interest in the scene to beginidentifying any brand identifier(s) included therein. However, if theuser indicates that all regions of interest have been identified,control proceeds to block 737 of FIG. 7C at which a tracker function isinitiated for each newly marked region of interest. As discussed above,a tracker function uses the marked region(s) of interest as atemplate(s) to track the corresponding region(s) of interest in theadjacent image frames comprising the current detected scene. After theprocessing at block 737 completes, the media stream 160 is restartedafter having been stopped (e.g., paused) (block 738). The example GUI152 then provides the scene and region(s) of interest to, for example,the example brand recognizer 254 to enable updating of brand identifiercharacteristics, and/or collection and/or calculation of brand exposureinformation related to the identified brand identifier(s) (block 740).Execution of the example machine accessible instructions 700 then ends.

Turning to FIG. 8A, execution of the example machine executableinstructions 800 begins with a brand recognizer, such as the examplebrand recognizer 254, receiving a scene, the scene's classification andone or more regions of interest from, for example, the example scenerecognizer 252, the example GUI 152, the processing at block 621 of FIG.6B, and/or the processing at block 728 of FIG. 7C (block 801). Theexample brand recognizer 254 then obtains the next region of interest tobe analyzed in the current scene from the information received at block801 (block 802). The example brand recognizer 254 then compares theregion of interest to one or more expected reference brand identifiertemplates (e.g., corresponding to a reference scene matching the currentscene) having, for example, one or more expected locations, sizes,orientations, etc., to determine which reference brand identifiermatches the region of interest (block 804). An example brand identifiermatching technique based on template matching that may be used toimplement the processing at block 804 is discussed above in connectionwith FIG. 4. Additionally, if the scene classification received at block801 indicates that the scene is a new scene, the comparison parametersof the brand identifier matching technique employed at block 804 may berelaxed to return brand identifiers that are similar to, but that do notnecessarily match, the compared region of interest.

Next, the example brand recognizer 254 returns the reference brandidentifier matching (or which closely matches) the region of interestbeing examined (block 806). Then, if any region of interest has not beenanalyzed for brand exposure reporting (block 808), control returns toblock 802 to process the next region of interest. If, however, allregions of interest have been analyzed (block 808), execution of theexample machine accessible instructions 800 then ends.

Turning to FIG. 8B, execution of the example machine executableinstructions 850 begins with a brand recognizer, such as the examplebrand recognizer 254, receiving information regarding one or moreregions of interest and one or more respective brand identifiersdetected therein from, for example, the example scene recognizer 252,the example GUI 152, the processing at block 625 of FIG. 6B, and/or theprocessing at block 734 of FIG. 7C (block 852). The example brandrecognizer 254 then obtains the next detected brand identifier to beprocessed from the one or more detected brand identifiers received atblock 852 (block 854). Next, one or more databases (e.g., the learnedknowledge database 264 FIG. 2, the brand library 266 of FIG. 2, etc.)are queried for information regarding the detected brand identifier(block 856). The brand identifier data may include, but is not limitedto, internal identifiers, names of entities (e.g., corporations,individuals, etc.) owning the brands associated with the brandidentifiers, brand names, product names, service names, etc.

Next, characteristics of a brand identifier detected in the region ofinterest in the scene are obtained from the information received atblock 852 (block 858). Next, the example brand recognizer 254 obtainsthe characteristics of the reference brand identifier corresponding tothe detected brand identifier and compares the detected brandidentifier's characteristics with the reference brand identifier'scharacteristics (block 860). The characteristics of the brand identifiermay include, but are not limited to, location, size, texture, color,quality, duration of exposure, etc. The comparison at block 860 allowsthe example brand recognizer 254 to detect and/or report changes in thecharacteristics of brand identifiers over time. After the processing atblock 860 completes, the identification information retrieved at block856 for the detected brand identifier, the detected brand identifier'scharacteristics determined at block 858 and/or the changes in the brandidentifier detected at block 860 are stored in one or more databases(e.g., such as the brand exposure database 155 of FIG. 1) for reportingand/or further analysis (block 812). Then, if any region of interest hasnot yet been analyzed for brand exposure reporting (block 814), controlreturns to block 854 to process the next region of interest. If allregions of interest have been analyzed, execution of the example machineaccessible instructions 850 then ends.

FIG. 9 is a schematic diagram of an example processor platform 900capable of implementing the apparatus and methods disclosed herein. Theexample processor platform 900 can be, for example, a server, a personalcomputer, a personal digital assistant (PDA), an Internet appliance, aDVD player, a CD player, a digital video recorder, a personal videorecorder, a set top box, or any other type of computing device.

The processor platform 900 of the example of FIG. 9 includes at leastone general purpose programmable processor 905. The processor 905executes coded instructions 910 and/or 912 present in main memory of theprocessor 905 (e.g., within a RAM 915 and/or a ROM 920). The processor905 may be any type of processing unit, such as a processor core, aprocessor and/or a microcontroller. The processor 905 may execute, amongother things, the example machine accessible instructions of FIGS.6A-6B, 7A-7C, and/or 8A-8B to implement any, all or at least portions ofthe example brand exposure monitor 150, the example GUI 152, the examplescene recognizer 252, the example brand recognizer 254, etc.

The processor 905 is in communication with the main memory (including aROM 920 and/or the RAM 915) via a bus 925. The RAM 915 may beimplemented by DRAM, SDRAM, and/or any other type of RAM device, and ROMmay be implemented by flash memory and/or any other desired type ofmemory device. Access to the memory 915 and 920 may be controlled by amemory controller (not shown). The RAM 915 and/or any other storagedevice(s) included in the example processor platform 900 may be used tostore and/or implement, for example, the example brand exposure database155, the example scene database 262, the example learned knowledgedatabase 264 and/or the example brand library 266.

The processor platform 900 also includes an interface circuit 930. Theinterface circuit 930 may be implemented by any type of interfacestandard, such as a USB interface, a Bluetooth interface, an externalmemory interface, serial port, general purpose input/output, etc. One ormore input devices 935 and one or more output devices 940 are connectedto the interface circuit 930. For example, the interface circuit 930 maybe coupled to an appropriate input device 935 to receive the examplemedia stream 160. Additionally or alternatively, the interface circuit930 may be coupled to an appropriate output device 940 to implement theoutput device 270 and/or the GUI 152.

The processor platform 900 also includes one or more mass storagedevices 945 for storing software and data. Examples of such mass storagedevices 945 include floppy disk drives, hard drive disks, compact diskdrives and digital versatile disk (DVD) drives. The mass storage device945 may implement for example, the example brand exposure database 155,the example scene database 262, the example learned knowledge database264 and/or the example brand library 266.

As an alternative to implementing the methods and/or apparatus describedherein in a system such as the device of FIG. 9, the methods and orapparatus described herein may be embedded in a structure such as aprocessor and/or an ASIC (application specific integrated circuit).

Finally, although certain example methods, apparatus and articles ofmanufacture have been described herein, the scope of coverage of thispatent is not limited thereto. On the contrary, this patent covers allmethods, apparatus and articles of manufacture fairly falling within thescope of the appended claims either literally or under the doctrine ofequivalents.

What is claimed is:
 1. A method to mark a region in a graphicalpresentation, the method comprising: identifying, with a processor, aplurality of user-selected points selected in the graphical presentationwith an input device; determining, with the processor, that theplurality of user-selected points defines a boundary of the region;responsive to a first user input received prior to or during selectionof the plurality of user-selected points with the input device,comparing, with the processor, a plurality of stored templates with theplurality of user-selected points to identify a first one of the storedtemplates to mark the region; and displaying, in the graphicalpresentation, the first one of the stored templates identified by theprocessor based on the comparing of the plurality of the storedtemplates with the plurality of user-selected points, prior to the firstone of the stored templates being accepted for marking the region.
 2. Amethod as defined in claim 1, further including: identifying asubsequent user-selected point in the graphical presentation selectedafter the first one of the stored templates is identified to mark theregion; reshaping at least a portion of a depiction of the first one ofthe stored templates in the graphical presentation based on a locationof the subsequent user-selected point to determine a reshaped depictionof the first one of the stored templates; and marking the region in thegraphical presentation based on the reshaped depiction of the first oneof the stored templates.
 3. A method as defined in claim 1, wherein thecomparing of the plurality of stored templates with the plurality ofuser-selected points occurs in response to a key press detected prior toor during selection of a first one of the plurality of user-selectedpoints.
 4. A method as defined in claim 1, wherein the comparing of theplurality of stored templates with the plurality of user-selected pointsoccurs in response to a selection of a virtual button on a graphicaldisplay detected prior to or during selection of a first one of theplurality of user-selected points.
 5. A method as defined in claim 1,wherein the comparing of the plurality of stored templates with theplurality of user-selected points includes: determining normalizedcross-correlations of the plurality of user-selected points withrespective ones of the plurality of stored templates; and identifying arespective one of the plurality of stored templates associated with alargest normalized cross-correlation to be the first one of the storedtemplates.
 6. A method as defined in claim 1, wherein the first one ofthe stored templates is identified by the comparing when not allvertices of a polygon bounding the region are included in the pluralityof user-selected points.
 7. A computer readable storage device orstorage disk comprising computer readable instructions which, whenexecuted, cause a computer to at least: identify a plurality ofuser-selected points selected in a graphical presentation with an inputdevice; determine that the plurality of user-selected points defines aboundary of a region to be marked in the graphical presentation;responsive to a first user input received prior to or during selectionof the plurality of user-selected points, compare a plurality of storedtemplates with the plurality of user-selected points to identify a firstone of the stored templates to mark the region; and display, in thegraphical presentation, the first one of the stored templates identifiedby the computer based on the comparison of the plurality of the storedtemplates with the plurality of user-selected points, prior to the firstone of the stored templates being accepted for marking the region.
 8. Astorage device or storage disk as defined in claim 7, wherein theinstructions, when executed, further cause the computer to: identify asubsequent user-selected point in the graphical presentation selectedafter the first one of the stored templates is identified to mark theregion; reshape at least a portion of a depiction of the first one ofthe stored templates in the graphical presentation based on a locationof the subsequent user-selected point to determine a reshaped depictionof the first one of the stored templates; and mark the region in thegraphical presentation based on the reshaped depiction of the first oneof the stored templates.
 9. A storage device or storage disk as definedin claim 7, wherein the instructions, when executed, cause the computerto compare the plurality of stored templates with the plurality ofuser-selected points in response to a key press detected prior to orduring selection of a first one of the plurality of user-selectedpoints.
 10. A storage device or storage disk as defined in claim 7,wherein the instructions, when executed, cause the computer to comparethe plurality of stored templates with the plurality of user-selectedpoints in response to a selection of a virtual button on a graphicaldisplay detected prior to or during selection of a first one of theplurality of user-selected points.
 11. A storage device or storage diskas defined in claim 7, wherein to compare the plurality of storedtemplates with the plurality of user-selected points, the instructions,when executed, cause the computer to: determine normalizedcross-correlations of the plurality of user-selected points withrespective ones of the plurality of stored templates; and identify arespective one of the plurality of stored templates associated with alargest normalized cross-correlation to be the first one of the storedtemplates.
 12. A storage device or storage disk as defined in claim 7,wherein the first one of the stored templates is identified when lessthan all vertices of a polygon bounding the region are included in theplurality of user-selected points.
 13. An apparatus to mark a region ina graphical presentation, the apparatus comprising: an input device; aprocessor to: identify a plurality of user-selected points selected withthe input device in the graphical presentation; determine that theplurality of user-selected points defines a boundary of the region;responsive to a first user input received prior to or during selectionof the plurality of user-selected points, compare a plurality of storedtemplates with the plurality of user-selected points to identify a firstone of the stored templates to mark the region; and display, in thegraphical presentation, the first one of the stored templates identifiedby the processor based on the comparison of the plurality of the storedtemplates with the plurality of user-selected points, prior to the firstone of the stored templates being accepted for marking the region; andmemory to store the plurality of stored templates.
 14. An apparatus asdefined in claim 13, wherein the processor is further to: identify asubsequent user-selected point in the graphical presentation selectedafter the first one of the stored templates is identified to mark theregion; reshape at least a portion of a depiction of the first one ofthe stored templates in the graphical presentation based on a locationof the subsequent user-selected point to determine a reshaped depictionof the first one of the stored templates; and mark the region in thegraphical presentation based on the reshaped depiction of the first oneof the stored templates.
 15. An apparatus as defined in claim 13,wherein the processor is to compare the plurality of stored templateswith the plurality of user-selected points in response to a key pressdetected prior to or during selection of a first one of theuser-selected plurality of points.
 16. An apparatus as defined in claim13, wherein the processor is to compare the plurality of storedtemplates with the plurality of user-selected points in response to aselection of a virtual button detected prior to or during selection of afirst one of the plurality of user-selected points, the virtual buttonincluded in a graphical user interface presented by the processor. 17.An apparatus as defined in claim 13, wherein to compare the plurality ofstored templates with the plurality of user-selected points, theprocessor is to: determine normalized cross-correlations of theplurality of user-selected points with respective ones of the pluralityof stored templates; and identify a respective one of the plurality ofstored templates associated with a largest normalized cross-correlationto be the first one of the stored templates to mark the region.
 18. Anapparatus as defined in claim 13, wherein the processor is to identifythe first one of the stored templates when not all vertices of a polygonbounding the region are included in the plurality of user-selectedpoints.
 19. A method as defined in claim 1, wherein the region is afirst region, the plurality of user-selected points is a first pluralityof user-selected points, and further including: comparing the pluralityof stored templates with a second plurality of user-selected points toidentify a second one of the stored templates to mark a second region inthe graphical presentation; and responsive to a second user input,automatically determining a shape bounding a first set of pixels in thegraphical presentation marked by the first one of the stored templatesand a second set of pixels in the graphical presentation marked by thesecond one of the stored templates to mark a third region in thegraphical presentation corresponding to a combination of the firstregion and the second region.
 20. A storage device or storage disk asdefined in claim 7, wherein the region is a first region, the pluralityof user-selected points is a first plurality of user-selected points,and the instructions, when executed, further cause the computer to:compare the plurality of stored templates with a second plurality ofuser-selected points to identify a second one of the stored templates tomark a second region in the graphical presentation; and responsive to asecond user input, automatically determine a shape bounding a first setof pixels in the graphical presentation marked by the first one of thestored templates and a second set of pixels in the graphicalpresentation marked by the second one of the stored templates to mark athird region in the graphical presentation corresponding to acombination of the first region and the second region.
 21. An apparatusas defined in claim 13, wherein the region is a first region, theplurality of user-selected points is a first plurality of user-selectedpoints, and the processor is further to: compare the plurality of storedtemplates with a second plurality of user-selected points to identify asecond one of the stored templates to mark a second region in thegraphical presentation; and responsive to a second user input,automatically determine a shape bounding a first set of pixels in thegraphical presentation marked by the first one of the stored templatesand a second set of pixels in the graphical presentation marked by thesecond one of the stored templates to mark a third region in thegraphical presentation corresponding to a combination of the firstregion and the second region.